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objective of this research is to analyze whether a relationship exists between source 
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System (CPARS) ratings, Earned Value Management (EVM) outcomes, and PALT. 
Multivariate and univariate analysis of covariance (MANCOVA and 
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I. 


INTRODUCTION 


According to the Federal Acquisition Regulation (FAR), there is an obligation for 
any government acquisition team to “deliver on a timely basis the best value product or 
service to the customer, while maintaining the public’s trust and fulfilling public policy 
objectives” (FAR 1.102[a]). Acquisition professionals are given significant discretion on 
exactly how they will meet that mandate. As acquisition teams develop their strategy, one 
of the most critical aspects to consider is the source selection methodology that will 
derive the best value outcome for the government. 

The FAR outlines a range of source selection methodologies in the best value 
continuum from the lowest price technically acceptable (LPTA) method at one end to the 
highest technically rated offeror (HTRO) at the other (FAR 15.101-1). Because the FAR 
mandates that “price or cost to the Government shall be evaluated in every source 
selection” (FAR 15.304(c)[ 1 ]), the tradeoff method is utilized most often in lieu of 
HTRO. The source selection methodology to be utilized to achieve best value depends on 
the complexity of the requirement as well as the importance of price or cost and non-cost 
factors to be evaluated (FAR 15.101). According to Calisti (2015), “industry contends, 
overusing LPTA in the long haul will erode the DOD technological edge through low- 
cost/low-performance solutions; cause perfonnance innovators to depart the market and 
reduce the quality of goods and services provided” (p. 17). The government on the other 
hand believes that if LPTA is utilized properly by ensuring that the technical 
acceptability is at the appropriate level, it can achieve best value more efficiently 
(Kendall, 2012). This research will analyze contracts that were awarded under both 
LPTA and tradeoff methods to determine whether a relationship exists between 
performance outcomes and the source selection method at the Air Force Materiel 
Command. 

A. PROBLEM STATEMENT 

There is an anecdotal belief that the government receives a higher quality product 
or superior level of perfonnance from source selections conducted through the tradeoff 


1 



method versus LPTA. However, there is no empirical evidence to prove that this belief is 
true. There is not a significant body of research to validate whether contractor 
performance outcomes from tradeoff selection methods result in better value to the 
government and that outcomes from LPTA methods result in inferior quality. 

B. PURPOSE 

This research is aimed at determining whether there is a relationship between the 
source selection method (LPTA or tradeoff) and the subsequent outcome of the contract. 
Understanding this relationship will help the Air Force better prepare for future 
acquisitions. The ability to select the proper source selection method for each acquisition 
scenario will ensure that the right amount of resources get committed to each project 
which reduces waste in the acquisition process. 

C. RESEARCH OBJECTIVES AND QUESTIONS 

The objective of this research is to analyze whether a relationship exists between 
source selection methods (LPTA or tradeoff) and the level of resulting perfonnance 
outcomes. The contractor’s performance results shall be obtained from the Contractor 
Performance Assessment Reporting System (CPARS) and Earned Value Management 
System (EVMS). The following questions form the basis for this research: 

1. What is the relationship between a particular source selection method and 
the resulting performance outcome of the contractor? 

2. What is the relationship between a particular source selection method and 
the PALT? 

i. If there is a positive relationship, can the amount of additional 
resources utilized for that particular source selection method be 
quantified in terms of man-hours or in dollars? 

ii. Does the resulting performance outcome merit the use of that 
particular source selection method? 

D. METHODOLOGY 

The methodology for this research include a literature review of certain aspects of 
the federal contracting process which impacts decisions on source selection methods, the 
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best value continuum, source selection procedures, controversial issues over the use of 
LPTA, and the need for source selection training for federal acquisition professionals. 
Data collection was limited to contracts awarded within the Air Force Life Cycle 
Management Center (AFLCMC) Directorates located at Wright-Patterson AFB in 
Dayton, OH. The contracts reviewed for this research were selected based on an award 
value in excess of $ 1 million which is the minimum threshold for CPARS reporting, an 
award date between fiscal years 2010 and 2013, and competition using either LPTA or 
tradeoff source selection methods. The Federal Procurement Data System-Next 
Generation (FPDS-NG) was used to identify a list of contracts awarded between 2010 
and 2013. The AFLCMC Directorates were then asked to identify and physically provide 
all competitively awarded contracts from the list to include source selection material to 
be analyzed by the research team. 

E. BENEFITS 

This research seeks to identify whether a particular source selection method 
produces better contractor performance outcomes than another as well as to quantify the 
government resources needed to conduct that particular source selection method. The 
results of this research will provide federal acquisition professionals with better insight 
on what outcomes to expect as well as the level of resources required to pursue these 
source selection methods. These insights will help in future acquisition planning to 
reduce waste, streamline the acquisition process, and achieve best value. 

F. LIMITATIONS 

There are many limitations to this research which impacts the applicability of the 
results across the DOD. First, the data is only limited to Air Force contracts from one 
particular Major Command (MAJCOM) and only one contracting center within that 
MAJCOM. Moreover, due to time and manpower constraints, it was impossible to collect 
a significant amount of data. Finally, the data collected only contains contracts awarded 
from 2010 to 2013, which leaves a vast number of contracts that would otherwise qualify 
for this study unobserved. As more studies of this nature are conducted across other 
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installations and agencies, the findings will have even greater significance for 
government acquisition professionals across the DOD. 


G. ORGANIZATION OF THE REPORT 

This report is organized into five chapters to include this introduction. The next 
chapter (Chapter II) provides a literature review on the federal contracting process, the 
best value continuum, source selection procedures, and a discussion of the controversy 
over increased utilization of the LPTA source selection method in the DOD. Chapter III 
discusses the methodology for this research to include background infonnation regarding 
the AFLCMC, the methods used to identify the data, and how the data was obtained. 
Chapter IV outlines the results and findings of the data analysis as well as answers the 
research questions. The final chapter (Chapter V) summarizes and concludes this research 
report with a recommendation for future research. 

H. SUMMARY 

This chapter examined the background infonnation on the basis of this research to 
include the problem statement, purpose, research objectives and questions, the 
methodology, as well as the benefits and limitations of this study. The next chapter is the 
literature review which provides an overview of statutes important to the government 
contracting process. Additionally, it presents the best value continuum, highlighting the 
LPTA and tradeoff source selection methods. Finally, the literature review examines the 
source selection procedures and discusses the controversial issues over the increased use 
of LPTA in the DOD. 
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II. LITERATURE REVIEW 


The purpose of this chapter is to provide an overview of general information on 
the federal contracting process, which influences decisions about the source selection 
methods to be used in federal procurement. Specifically, this chapter begins with a 
review of applicable statutes that guide the management of federal contracts and mandate 
competition as well as efficiency in the federal procurement process. Further, the chapter 
reviews the various source selection methods available in the best-value continuum, 
highlighting the important differences between LPT A and tradeoff. Additionally, this 
chapter examines the DOD source selection procedures, as well as efforts made to reform 
these procedures to maximize the concept of best value. The chapter concludes with a 
discussion of a number of controversial issues surrounding the increased use of LPT A 
and the need for robust source selection training for federal acquisition professionals. 

A. CONTRACT MANAGEMENT STATUTES AND REGULATIONS 

The regulatory agency responsible for “shaping the policies and practices federal 
agencies use to acquire the goods and services they need to carry out their 
responsibilities” (White House, n.d., OMB section para. 2) to the tune of approximately 
$530 billion each year is called the Office of Federal Procurement Policy (OFPP). The 
OFPP was established by Congress to “provide overall direction for government-wide 
procurement policies, regulations, and procedures and to promote economy, efficiency, 
and effectiveness in acquisition processes” (White House, n.d., OMB section para. 2). 
Important statutes provide the foundation for critical aspects of government contracting, 
particularly decisions about source selection methods to be utilized. These include the 
Competition in Contracting Act (CICA) and the Federal Acquisition Streamlining Act 
(FASA). 

1. Competition in Contracting Act 

The mandate for competition is the key factor that forces agencies to conduct 

source selections for procurement actions. Competition not only helps the government 

achieve best value at a fair and reasonable price, but it is also vital to the small business 
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program, the defense industry, and the economy. The CICA, enacted in 1984, requires 
that all contemplated actions leading to the award of contracts must be entered into a full 
and open competition (Manuel, 2011). FAR 6.101 prescribes that 

contracting officers shall provide for full and open competition through 
use of the competitive procedures(s) contained in this subpart that are best 
suited to the circumstances of the contract action and consistent with the 
need to fulfill the government’s requirements efficiently. (FAR 6.101 [b]) 

While the promotion of maximum competition is preferred, there are 
circumstances that prevent the government from being able to compete certain types of 
requirements or when a sole source environment is more advantageous to the 
government. In such cases, the FAR “prescribes policies and procedures, and identifies 
the statutory authorities, for contracting without providing for full and open competition” 
(FAR 6.3). The source selection methods and procedures for competitive awards are 
discussed in more detail later in this chapter. 

2. Federal Acquisition Streamlining Act 

Another notable statute that influences the decision making process for source 
selection methods by mandating efficiencies in the federal procurement practices is the 
FASA. Due to the size and complexity of the federal acquisition system, Congress 
enacted the Federal Acquisition Streamlining Act of 1994 with the goal of achieving 
efficiencies in the acquisition process (Drelicharz, 1994). According to Drelicharz 
(1994), the intent of FASA “is to develop a more equitable balance between government- 
unique requirements and the need to lower the government’s cost of doing business” 
(p. 12). Drelicharz (1994) also reports that 

the act addresses these objectives through an emphasis on increasing 
government-wide reliance on the use of commercial practices, goods and 
services; streamlining the rules and regulations that govern high-volume 
contracting activities, which represent an overall low-dollar expenditure; 
and increasing the selling opportunities for small businesses, (p. 12) 

Drelicharz (1994) explained a few of the noteworthy changes brought about by 
the enactment of the FASA include the following: 
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• increasing the simplified acquisition threshold (SAT) from $25,000 to 

$ 100 , 000 ; 

• enabling government purchase of commercial products on commercial 
terms; 

• standardizing the procurement system throughout all federal agencies; 
providing notices and debriefs to unsuccessful bidders upon request; 

• providing authority to the General Services Board of Contract Appeals 
(GSBCA) to “dismiss frivolous protests” and “invoke procedural sanctions 
where a person makes a bad-faith protests” (Drelicharz, 1994, P.15); and 

• requiring the establishment of a Federal Acquisition Computer Network 
(FACNET) as a way to “dramatically streamline the federal purchasing 
process” (Drelicharz, 1994, p. 15). 

The change that directly impacts source selection methods is the new SAT. The 
increased SAT allows contracting officers to use simplified procedures for competitive 
awards up to $100,000 under FAR 13 in lieu of formal source selection procedures under 
FAR 15 (Barry, 1995). These statutes not only impact decisions pertaining to source 
selection methods but also affect how the contract management processes are performed. 

B. CONTRACT MANAGEMENT PROCESS 

The contract management process is an overarching endeavor that includes a 
number of activities to include developing a requirement, awarding the contract, and 
managing relationships with the contractor after award. In his book titled World Class 
Contracting, Garrett defines the contract management process as “the art and science of 
managing a contractual agreement throughout the contracting process” (Garrett, 2015, 
p. 15). Understanding this process provides an important foundation to the factors that 
impact competition, source selection, and best value. 

According to Garrett (2015), there are three phases and six primary steps in the 
contract management process, as shown in Figure 1. 
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Figure 1. Contract Management Process 



Adapted from: Garrett, G. A. (2015). World class contracting (6th ed.). Chicago, IL: 
Wolters Kluwer. 


For purposes of this research, we will only focus on the processes of the buyer 
since we will be analyzing Air Force procurement data as the buying agency. The 
following paragraphs will briefly describe the three phases along with the purpose of 
each step and the activities generally performed during those steps. 

1. Pre-award Phase 

According to Garrett (2015), the pre-award phase is mostly concerned with 
planning for the acquisition, determining what the government needs, what is available in 
the market, making a decision to provide solutions in-house or outsourcing, and soliciting 
the market to acquire the needs. Garrett (2015) identifies procurement planning, 
solicitation planning, and solicitation as the three important steps to focus on during this 
phase because “this phase is vital in creating successful business relationships” (Garrett, 
2015, p. 17). Since using contracts to procure needs involve inherent risks associated with 
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expectation management of both contracting parties, it is important to perform each step 
of the pre-award phase with due diligence to minimize this risk (Garrett, 2015). 

a. Step 1: Procurement Planning 

The most important decision to be made during the procurement planning step 
according to Garrett (2015) is the “make-or-buy decision or outsourcing decision” 
(Garrett, 2015, p. 20). The primary focus of this step is to determine what the needs of the 
organization are, when it must be filled, and how (Garrett, 2015). The FAR defines 
acquisition planning as “the process by which the efforts of all personnel responsible for 
an acquisition are coordinated and integrated through a comprehensive plan for fulfilling 
the agency need in a timely manner and at a reasonable cost” (FAR 2.101). Some of the 
activities performed during this step include defining the needs of the organization, 
analyzing internal resources and constraints, market intelligence, and risk analysis 
(Garrett, 2015). 

Market intelligence and market research are critical aspects of procurement 
planning for many reasons. For one thing, they help identify whether the requirement is a 
commercial need. Additionally, they identify potential companies interested in the 
requirement which is necessary to make decisions about small business and or 
socioeconomic set-asides. This information is also necessary to determine the 
procurement method as well as the source selection method such as LPTA or tradeoff. 
Hosting industry days or posting sources sought notices on the Government Point of 
Entry (GPE) are great ways to obtain industry information. 

Risk analysis is another important activity to be conducted during the 
procurement planning step (Garrett, 2015). In order to minimize the various types of risks 
associated with contracting, “the buyer must identify the sources of uncertainty about 
contract performance and the risks associated with those uncertainties to write a contract 
that will address those risks and fairly distribute them between buyer and seller” (Garrett, 
2015, p. 73). Risk analysis is a critical activity in the detennination of the contract type. 
Cost, schedule, and performance are just a few examples of risk areas that should be 
analyzed. There are various methods to avoid, reduce, or transfer risk; however, there are 
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also tradeoffs or costs associated with them as well. For example, cost risk could be 
transferred to the contractor by using a firm fixed price type of contract. In turn, the 
contractor will build the cost of his or her risk into the offering price. 

The personnel involved in performing such activities for procurement planning 
should consist of a multi-functional team. In government acquisitions, a typical 
acquisition team includes members from the requiring organization, contracting, legal, 
and finance. The results of these activities should be used to support the critical decision 
of sourcing within the organization or acquiring the needs from outside (Garrett, 2015). 
According to Garrett (2015), this make or buy decision is extremely important because 
the decisions can impact an organization’s strategy and may be critical. The activities 
conducted during the procurement planning process provide for an important foundation 
for the next step in the preaward phase. 

b. Step 2: Solicitation Planning 

With a fully developed description of what is required and the decision to procure, 
the next step in the pre-award phase is solicitation planning. The purpose of this step is to 
develop the documentation necessary to solicit the requirement (Garrett, 2015). 
Depending on the type of requirement, these documents will describe what the 
government needs and how the work should be performed or delivered. A draft 
solicitation or request for proposal (RFP) outlines the quantity and description of the 
items or services to be performed, the contract specifications such as a statement of work 
(SOW), applicable clauses, instructions on how offerors should respond, and how 
proposals will be evaluated. The criteria for evaluating proposals may vary depending on 
the procurement method selected as well as the source selection method to be utilized. 

c. Step 3: Solicitation 

With the proper documents such as the SOW and the RFP built, the next step in 

the pre-award phase is the solicitation. The purpose of this step “consists of obtaining 

information (bids and proposals) from prospective sellers on how project needs can be 

met” (Garrett, 2015, p. 78). During solicitation, a pre-bid conference may be held and the 

requirement advertised, among other activities (Garrett, 2015). A pre-bid conference can 
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be used to ensure that interested vendors fully understand the requirement from both a 
technical as well as contractual stand point, and to provide clarifications or answer 
questions which can lead to amendments to the RFP (Garrett, 2015). In a competitive 
solicitation, an RFP is commonly advertised through the GPE at www.fbo.gov. The pre¬ 
award phase ends with the receipt of proposals or bids submitted by vendors on a timely 
basis as a result of the solicitation process (Garrett, 2015). 

2. Award Phase 

With all the proposals in house and ready for evaluations, the next phase of the 
contract management process, the award phase is ready to begin. Source selection is the 
only step of this phase where many critical activities will be performed. These activities 
are vital to the successful fulfillment of agency needs through a contract (Garrett, 2015). 
The government’s goal during this phase is to make an award to an offeror that can meet 
the buyer’s needs at the best value. The DOD Source Selection Procedures are the most 
utilized approach to making such awards in government acquisitions. 

a. Step 4: Source Selection 

Source Selection is the process of evaluating the proposals received in order to 
select the right offeror to fill the needs. For competitive awards, the methods generally 
used to make selections are LPTA or tradeoff depending on how well the requirement is 
defined (FAR 15.101). According to Garrett (2015), “the process used to accomplish this 
varies dramatically depending on the company, the products or services involved, the 
complexity of the procurement, and many other factors” (p. 138). The key to evaluating 
proposals successfully in government acquisitions is to ensure that the evaluation factors 
presented in Section M of the solicitation is followed exactly as prescribed. A few of the 
commonly evaluated factors in government requirements are price or cost, technical 
approach, and past performance. Tools such as contract negotiation, weighting systems, 
screening systems, and independent estimates may be used to evaluate these factors 
during the source selection process to achieve the optimum outcome, which is a best 
value contract (Garrett, 2015). 
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3. 


Post Award Phase 


An established contract with a selected vendor ready to perform marks the 
beginning of the post-award phase of the contract management process. This final phase 
involves two steps to ensure that ah terms and conditions of the contract are carried out 
properly and that the contract gets closure upon performance completion or termination 
(Garrett, 2015). Ah parties involved in the contract will need to stay in constant 
communication in order to achieve the desired outcomes of the contract through 
performance, progress, and contract compliance monitoring (Garrett, 2015). 

a. Step 5: Contract Administration 

Contract administration, according to Garrett (2015), “is the process of ensuring 
that each party’s perfonnance meets contractual requirements” (p. 140). Success depends 
on full comprehension of the requirements and responsibilities of both parties; therefore, 
the contract should be carefully studied by ah affected personnel and organizations on 
both sides (Garrett, 2015). The contract administration process involves many activities 
with a variety of tools and techniques to help perform these activities effectively. In their 
book titled Management of Defense Acquisition Projects, Rendon and Snider (2008) offer 
four techniques typically utilized in the government contract administration process 
which includes: 

1. Conducting a pre-performance conference, 

2. Monitoring the contractor’s work results, 

3. Measuring contractors performance, and 

4. Managing the change-control process (Rendon & Snider, 2008, p. 176). 

These techniques are designed to ensure that ah parties are standing on common 

ground as it relates to the requirements, responsibilities, processes, and protocols that 
both parties will follow to resolve any conflicts or disputes until the contract is ready for 
closeout. 
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b. Step 6: Contract Closeout 

The sixth and final step of the contract management process deals with closeout 
or termination. According to Garrett (2015), “a contract can end in one of three ways: 
successful performance, mutual agreement, or breach of contract” (p. 159). Regardless of 
whether a contract ends due to successful completion or tennination, the closeout process 
must ensure that all administrative matters such as final payments or settlements are paid 
and that documents necessary to closeout the contract are received (Garrett, 2015). In 
government acquisitions, FAR 4.804 outlines the responsibilities for carrying out the 
closeout process which primarily focuses on the acceptance of the good or service, 
payment to the contractor, and the statutory timelines for conducting the closeout. 

The contract management process shows the important roles each of the phases 
and steps play in government acquisitions. Most importantly, this process clearly shows 
that best value cannot be achieved through just one single phase or step. The extensive 
leg work performed prior to and after the source selection phase of the entire contract 
management process influences the objectives of achieving best value. The next section 
provides an overview of the best value concept and the best value continuum as 
prescribed by the FAR. 

C. BEST VALUE AND BEST VALUE CONTINUUM 

The underlying goal of every contract in the federal government should be to 
acquire a product or service that provides the “best value.” The FAR defines best value as 
an “expected outcome of an acquisition that, in the government’s estimation, provides the 
greatest overall benefit in response to the requirement” (FAR 2.101). It is important to 
note that best value is not always achieved by awarding to the lowest priced bidder. 
Certainly, there are instances where price should be, and is, the overarching factor that 
determines the winning bid. Conversely, there are times when factors other than price or 
cost, such as a technical approach to complex requirements or exceptional past 
performance records, garner additional consideration by the acquisition team as a way to 
obtain the best product or service. 
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In order to address the myriad degrees of complexity in government requirements, 
the FAR lays out distinct methods by which best value can be achieved through the use of 
the best value continuum. The interaction of price and non-cost factors that affect the 
source selection methods to be utilized according to the best value continuum is clearly 
described in the FAR: 

An agency can obtain best value in negotiated acquisitions by using any 
one or a combination of source selection approaches. In different types of 
acquisitions, the relative importance of cost or price may vary. For 
example, in acquisitions where the requirement is clearly definable and the 
risk of unsuccessful contract performance is minimal, cost or price may 
play a dominant role in source selection. The less definitive the 
requirement, the more development work required, or the greater the 
performance risk, the more technical or past performance considerations 
may play a dominant role in source selection. (FAR 15.101) 

According to the best value continuum, as seen in Figure 2, the relative 
importance of price as compared to non-cost factors determines whether the LPTA or 
tradeoff method should be used. 


Figure 2. The Best Value Continuum 


Across the Best value Continuum, the relative importance of cost or price may vary. 


FM 15.101 


Tradeoff Process 


Sum of Non-$ factors Significantly < $ 


Sum of Non-$ foe tors approx. - 5 


Sum of Non-$ foctors Significantly > $ 



LPTA is used at the endpoint of the 
Best Value Continuum where cost 
or price has maximum weight and 
all other factors are pass/fail. 


Per statute, quality of product or seme sholl be addressed in every source selection. isjot 

Factors OtherThan Cost or Price 

2.3.1.2 describes the evaluation of quality 
of product or service via the factors of: 

• Technical 

Cost or Price Factor J ■ Past Performance 

Per stotute, cost or price IS) sholl be evaluated in every source selection. fm isaor 


Best Value Continuum 


Adapted from: Best value continuum, (n.d.). In ACQuipedia. Retrieved from 
https://dap.dau.mil/acquipedia/Pages/ArticleDetails.aspx7akR5201f734-3bce-4c5f-a5a3- 

47551df77ea5 
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The LPT A method is at one end of the continuum with the heaviest emphasis on 
price or cost as the most important factor. The tradeoff method is a process that moves 
away from the LPTA method on the continuum as the sum of non-cost factors such as 
technical, past performance, and or small business plan becomes equally important or 
more important than price or cost. 

1. Lowest Price Technically Acceptable 

The ideal situation for using the LPTA method is for those requirements that are 
well defined and where “the solicitation provides the evaluation factors and significant 
subfactors that establish the requirements of acceptability” (Rendon & Snider, 2008, 
p. 174). One aspect that makes the use of LPTA desirable over tradeoff is the ease with 
which technical evaluations can be conducted. Since all technical factors are based on an 
“acceptable” or “unacceptable” basis, the award is simply made to the bidder offering the 
lowest price with an acceptable rating on all technical factors. 

The FAR states that an LPTA approach is acceptable only when the government 
believes that “best value is expected to result from selection of the technically acceptable 
proposal with the lowest evaluated price” (FAR 15.101-2). Considering this definition, 
the LPTA method is best suited for situations where the government would not realize 
any additional value by paying higher prices for non-cost factors that exceed minimum 
needs (OUSD[AT&L], 2011). This is primarily the case in acquisitions for non-complex 
or commercially available products or services. 

2. Tradeoff Process 

When contemplating complex acquisitions with a degree of risk that can impact 
cost, schedule, or performance, which is deemed harmful to the program, the best source 
selection method is the tradeoff. While this is considered a more resource-intensive 
approach to procuring goods and services, it allows the government to pay higher prices 
for reduced perfonnance risk, and/or greater technical capabilities, which may provide 
better overall value. According to the FAR, “a tradeoff process is appropriate when it 
may be in the best interest of the government to consider award to other than the lowest 

priced offeror or other than the highest technically rated offeror” (FAR 15.101-1). 
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The tradeoff method of source selection is a process that “allows the government 
the flexibility to award to an offeror anywhere on the best-value continuum between the 
lowest priced technically acceptable offeror and the highest technically rated offeror” 
(Rendon & Snider, 2008, p. 175). The FAR gives acquisition professionals a broad 
discretion of which evaluation factors and significant subfactors can be used in 
solicitations so long as these factors “are tailored to the acquisition,” “represent the key 
areas of importance and emphasis to be considered in the source selection decision,” and 
“support meaningful comparison and discrimination between and among competing 
proposals” (FAR 15.304). Non-cost factors that are typically used in the tradeoff process 
include, but are not limited to, adherence to the solicitation, superior technical approach, 
relevant and recent past perfonnance, small business subcontracting plan, professional 
qualification, risk reduction plan, and the ability to meet or exceed the schedule. 

D. DEPARTMENT OF DEFENSE SOURCE SELECTION PROCEDURES 

Regardless of the source selection method (LPTA or tradeoff) used to acquire 
goods and services, all competitive awards under FAR 15 must follow the DOD 
standardized Source Selection Procedures (DOD, 2011). In November 2008, John J. 
Young, the Undersecretary of Defense for Acquisition, Technology, and Logistics 
(USD[AT&L]), established the Source Selection Joint Analysis Team (JAT; 
USD[AT&L], 2011). The primary charge for the JAT was to “standardize the 
methodology and approach the Department uses to conduct competitively negotiated 
source selections” (USD[AT&L], 2011, p. 1). It took the JAT over two years to develop a 
robust, standardized approach to conducting source selections; the use of these 
procedures became mandatory across the DOD as of July 2011 (USD[AT&L], 2011). 

The Source Selection Procedures give acquisition teams a start-to-finish road map 
of the entire source selection process. It begins with a statement of its purpose, outlines 
the procedures, defines the roles and responsibilities of each source selection team 
member and provides a standardized listing of all documents that should be used during 
the process (DOD, 2011). When referenced properly, the DOD Source Selection 
Procedures should help agencies achieve successful source selection results, shield the 
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agencies from sustainable protests, and help achieve the goal “to ensure the Department’s 
source selection process delivers quality, timely products and services to the Warfighter 
and the Nation at the best value for the taxpayer” (DOD, 2011, p. 1). 

E. ACQUISITION REFORM 

Prior to the implementation of a standardized DOD source selection guide, the 
process for source selection and the concept of best value have gone through several 
iterations of acquisition reform throughout the years. Some of the earlier attempts started 
in 1981 through the Carlucci acquisition refonn initiatives and later through the FAS A, 
the FARA, and the Better Buying Power (BBP) initiatives (Fox, 2011). 

One of the early attempts to redefine and clarify the best value concept was 
through the FASA. As mentioned earlier, this act focuses on simplifying and refonning 
acquisition laws in order to achieve efficiencies in the acquisition process. A portion of 
this act emphasized the concept of best value, which allowed agencies to award contracts 
to higher-priced offerors based on a tradeoff of evaluation factors (FAR 15.101-1). The 
act, however, missed a couple of important issues according to Fox (2011), “for example, 
instead of giving procuring offices the freedom to use innovative methods and their own 
judgment to obtain the ‘best value’ for their organizations, the bill included new 
restrictions and oversight” (p. 166). Furthermore, “it provided no simplification or 
streamlining at all for contracts valued over $100,000” (p. 166). 

While the FASA was a good starting point for much needed acquisition reforms, 
there has been conflicting emphasis on the latitude given to contracting officers in source 
selections. With the passage of the FARA, which mandates “that full and open 
competition in government contracting be implemented in a way that is consistent with 
the need to fulfill efficiently the government’s requirements” (FARA, n.d., para. 1), 
contracting officers gained more discretion to “limit the number of proposals in the 
competitive range”, which promotes source selection efficiency. This latitude to limit the 
number of proposals to the highest rated bidders is deemed “one of the most significant 
changes in the government contracting area and one of the most controversial” (para. 2). 
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A renewed focus on source selections took place with efforts to standardize the 
process across the DOD. The idea to create the JAT was prompted by a report to 
Congress by the secretary of defense on the Acquisition Transformation Initiatives (OSD, 
2008). One of the recommendations was to “change existing source selection guidance to 
enhance communication to industry” in order to “improve everyone’s knowledge and 
understanding of the process and seeks to avoid protests and misunderstandings” (p. 21). 
Prior to the implementation of the DOD Source Selection Procedures, each service used 
agency-specific regulations and their own interpretations of FAR 15 procedures. This 
practice made it cumbersome and confusing for industry wishing to participate in source 
selections across the DOD. For example, a supplier participating in multiple source 
selections across different agencies and navigating the apparent and not so apparent 
differences in the way each agency conducted source selections could have been 
frustrated. The standardization of source selection procedures across the DOD is also 
important for obtaining better research data. With all agencies following the same 
procedures, the outcomes of each source selection can be documented and studied for 
potential trends. 

The Better Buying Power (BBP) initiatives beginning with 1.0 introduced in 2010 
began a new series of acquisition reforms focused on achieving efficiency and improving 
acquisition processes across the DOD (Carter, 2010). It was not until BBP 2.0 launched 
in 2012, however, that refonn initiatives addressed best value concepts. In his 
memorandum, Kendall (2012) outlines how this can be achieved: 

The Department needs to improve its ability to define the value to the 
Department of performance that is above minimum levels so that it can 
make appropriate source selections and so that industry can bid 
intelligently. This will spur innovation by providing a predictable basis by 
which companies can bid enhanced performance with the knowledge that 
any increased costs are within an acceptable range, (pp. 3-4) 

Recently, the DOD’s focus on “best value” was highlighted again in the release of 
BBP 3.0. This update sought to “provide clear and objective ‘best value’ definitions to 
industry” (Kendall, 2015b, p. 18). This effort is included within the “Incentivize 
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Innovation in Industry and Government” initiative, and Kendall (2015b) provides the 
following general guidance: 

This BBP 3.0 initiative builds on the work started in BBP 2.0 to provide 
industry with information on the value, in monetary terms, of higher levels 
of performance than minimally acceptable or threshold levels. Without 
this information, the default position will be to bid to the lowest 
acceptable level of performance. With this information, industry will 
know what the competitive effect of offering higher performance will be 
and can bid accordingly, (p. 18) 

Kendall espoused that the DOD would “focus on how to more effectively 
monetize best value and publicize relevant case studies” (Kendall, 2015b, p. 18). Specific 
actions include: (1) updating the DOD Source Selection Procedures by May 2015 to 
include the Best Value process manual; and (2) ensure that “best value” definitions are 
“objective and stated in monetary terms as much as possible” (Kendall, 2015b, p.18). 

In addition to best value concepts, the use of LPTA also entered the spotlight 
based on concerns from industry. Even before the use of LPTA was discussed in BBP 
2.0, the drumbeat from LPTA detractors was growing steadily louder. To address these 
growing concerns, Kendall followed up with a memorandum titled “Appropriate Use of 
Lowest Priced Technically Acceptable Source Selection Process and Associated Contract 
Type” (Kendall, 2015a). The bottom line of his memorandum is to address the fact that 
inappropriate use of particular source selection methods can undermine the government’s 
efforts to achieve best value (2015a). 

F. CONTROVERSY OVER USAGE 

Words such as “sequestration,” “economic recession,” and “budget deficit” have 
been main topics of headline news for several years. Young (2015) asserted that the DOD 
budget decreased steadily over the past several years since 2010 due to the winding down 
of conflicts in Iraq and Afghanistan and Congress passing the Budget Control Act of 
2011, which imposes caps on discretionary spending also known as sequester. According 
to Senator Jeff Sessions (2011) in a United States Senate Budget Committee report, the 
Budget Control Act is designed to cut defense spending by approximately 20% while 
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increasing non-defense spending by about 50%, “sequester affects defense more than 
other programs” (p. 1). 

In response to the national budget crisis, the fonner USD(AT&L), Ashton Carter 
(2010), emphasized the need to “do more without more” (p. 1) in Better Buying Power: 
Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending, a 
memorandum for all acquisition professionals. With the budget getting tighter each year, 
the DOD must find ways to achieve efficiencies across the board, especially within the 
realm of contracting processes and procedures. While it is unclear whether the budget 
constraints caused the increase in the DOD’s utilization of the LPTA source selection 
method, a 2014 GAO report indicated that “DOD increased its use of LPTA from fiscal 
year 2009 to fiscal year 2013 for contracts with obligations of $25 million or more” 
(p. 11). The DOD is also reported to be the top user of the LPTA procurement method, 
according to Langan (2013). 

This increase in the use of LPTA by government procurement offices is stirring 
up controversy between industry and government. As cited in Gansler and Lucyshyn 
(2013), “The National Defense Industrial Association (2012) named improper use of 
LPTA sourcing as one of its ‘top issues’ of 2012, while the Professional Services Council 
(PSC) called monitoring the frequency and misuse of LPTA a ‘2013 Policy Priority’” 
(p. v). According to the GAO (2014), the DOD is the biggest spender of the federal 
budget out of all other departments; it “obligated about $310 billion in fiscal year 2013 to 
acquire products and services needed to support its missions” (p. 1). This fact puts the 
DOD’s acquisition practices under a microscope, leading to high levels of scrutiny and 
criticism from all stakeholders, especially the defense industry and Congress. The issues 
surrounding the use of LPTA are making greater noise in the media and becoming the 
subject of leadership attention in recent years. 

1. Is LPTA Used Appropriately? 

With increased use of the LPTA method for contracts above $25 million, critics 
(mainly defense contractors) question whether such methods are being used 
appropriately. FAR 15.101 outlines the various methods within the Best Value 
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Continuum whereby agencies can choose “any one or a combination of source selection 
approaches” (FAR 15.101) to obtain best value. While the FAR outlines that “in different 
types of acquisitions, the relative importance of cost or price may vary” (15.101), the 
ultimate decision to use either the LPTA or tradeoff method depends on the contracting 
officer’s discretion. To provide further guidance on the issue, Kendall (2015a) pointed 
out in his DOD-wide memorandum that 

LPTA is the appropriate source selection process to apply only when there 
are well-defined requirements, the risk of unsuccessful contract 
performance is minimal, price is a significant factor in the source 
selection, and there is neither value, need, nor willingness to pay for 
higher performance, (p. 1) 

Additionally, the new DOD (2011) Source Selection Procedures suggest that 

LPTAs may be used in situations where the govermnent would not realize 
any value from a proposal exceeding the government’s minimum technical 
or performance requirements, often for acquisitions of commercial or non¬ 
complex services or supplies which are clearly defined and expected to be 
low risk. (p. A-l) 

Although there are published policies and guidance regarding the appropriate use of 
LPTA, Weckstein and Delgado (2012) claimed that “the legal framework on use of 
LPTA remains unworkably vague” (p. 2) and that there is no “firm standard for when 
LPTA can and cannot be used” (p. 3). For instance, it is up to the contracting officer’s 
discretion to decide the source selection method based on how complex and or risky he or 
she perceives a project to be. There is a lack of clear guidance about what makes a 
contract “complex” or “risky.” Weckstein and Delgado (2012) also argued that “there 
must be some limit to a contracting officer’s discretion, a point at which the decision to 
use LPTA sourcing becomes unreasonable in relation to agency needs and the 
government’s interest” (p. 3). 

Aside from a perceived lack of clarity in prescribing the proper use of various 
source selection methods, the DOD is criticized for using LPTA “where marginal product 
quality is a differentiator (such as medical products used to care for soldiers) and in the 
outcome-based services industry” (Crusius, 2015, para. 3). Referring to the government’s 
use of LPTA to procure such services, Crusius (2015) made a comparison by asking the 
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question “Would you like it if your children’s school hired its teachers on an LPTA 
basis?” (para. 3). Gansler and Lucyshyn (2013) also voiced their concern about the fact 
that “LPTA source selection has been expanded to the procurement of complex hardware 
and high-knowledge-content professional services” (p. 9). They stated that the use of 
LPTA “can be inappropriate at times, especially when the technology in question is 
complex or unprecedented, or when lives are at stake” (p. 4). Gansler and Lucyshyn 
(2013) also argued that high-end, complex services deserve a “thorough examination of 
‘trade-offs’ between cost and non-cost factors” (p. 2). Finally, Lohfeld (2012) remarked 
that “procurements get into trouble when the LPTA criteria is applied to technical and 
professional services bids because the work is complex, minimum acceptable technical 
and performance requirements are difficult to describe, and the consequences from failure 
can be considerable” (p. 1). 

Many of the articles, including Gansler and Lucyshyn’s (2013) study, point to the 
State Department’s personal security contract for the U.S. Embassy in Kabul, 
Afghanistan, as an example of a complex service procured through the LPTA method, 
which should have been acquired through tradeoff. According to the Commission on 
Wartime Contracting in Iraq and Afghanistan (CWC, 2009), 

unlike other federal agencies, the U.S. Department of State is forbidden by 
law to select anything but the lowest price and ‘technically acceptable’ 
offer when awarding contracts to protect its overseas buildings—even if 
this means passing up offers from firms offering higher quality and better 
experience. In contingency operations like those in Iraq and Afghanistan, 
this prohibition can have negative consequences for security, wartime 
mission objectives, and America’s image, (p. 1) 

Accordingly, the CWC (2009) reported that the winning bid was $110 million 
lower than its competitor and the contract was plagued with a “litany of complaints about 
poor contract performance and behavior by AGNA, the company’s in-country leadership 
team was slow to respond to problems and improperly performed their duties as a 
contingency contractor” (p. 4). The contractor’s performance deficiencies consisted of 
failing to provide the proper number of trained, qualified guards; inability of the guards 
to communicate with embassy personnel and visitors due to language barriers; and 
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unacceptable food services, inadequate documents, training, and unifonns for new 
employees, which were all requirements of the contract (CWC, 2009) 

High profile cases such as the State Department’s Kabul embassy example along 
with a few others such as the Air Force’s KC-X program and the Navy’s next generation 
enterprise network contract discussed by Gansler and Lucyshyn (2013), leave opponents 
of LPTA with a perception that the method is widely and inappropriately used across the 
DOD. However, the 2014 GAO report titled Defense Contracting: Factors DOD 
Considers When Choosing Best Value Processes Are Consistent with Guidance for 
Selected Acquisitions paints a much different picture. According to the results of this 
study, the “DOD used the tradeoff process most often in our sample to acquire services, 
including those related to construction projects, aircraft maintenance, and other support 
services, regardless of obligation amount” (GAO, 2014, p. 10). The report also claims 
that the LPTA method was used primarily for the purchase of “commercial products such 
as fuel” (p. 10) for contracts at or above $25 million. While it is true that the use of LPTA 
in DOD contracts increased over the past few years, its application is found to be 
consistent with guiding policies and procedures appropriate for the level of risk and 
complexity of the projects awarded under its method. 

2. Does LPTA Provide Best Value? 

The biggest issue with increased use of the LPTA method is that it is perceived by 
critics to be the cheapest, short-term focused solution, which is not a best value approach 
to procurement. According to Meagher (2012), “Lowest Price Technically Acceptable is 
commonly viewed as an alternative to best value” (para. 2) even though the FAR clearly 
outlines that it is one of the methods that can be used to achieve best value. The FAR 
specifically states that LPTA is appropriate “when best value is expected to result from 
the selection of the technically acceptable proposal with the lowest evaluated price” 
(FAR 15.101-2). Yet industry argues that best value cannot be achieved when price 
becomes the biggest discriminator, with technical capabilities and company experience 
seated in the back on a pass or fail basis. 
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With regard to best value, Gansler and Lucyshyn (2013) asserted that ignoring 
such important non-cost factors and focusing on lowest price could lead to awarding 
contracts to Finns with “fewer, or less experienced, employees; unreliable supply chains; 
or less effective quality control processes—factors that may not be evident in the 
proposal, and, even if they are, such factors are typically excluded from consideration 
under LPTA” (p. 2). According to Lohfeld (2015), “overemphasizing cost causes bidders 
to be much more aggressive at risk taking, and bidders will understate their cost of 
performance in an effort to be the lowest priced, technically acceptable bidder” (para. 4). 
Lohfeld (2015) also emphasized that “in the long run, expected cost savings are erased by 
cost overruns, contract modifications, show cause letters, and even contract terminations” 
(para. 4). When contractors are forced to compete on the basis of lowest price, Perkins 
(2014) opined, this drives contractors to hire the cheapest people they can possibly find 
and that “projects staffed via LPTA usually end up with the least expensive people 
meeting minimum levels of education, security clearance and so on. Such people are 
likely to have trouble with unexpected or complex problems” (para. 4). Most companies 
in the defense industry seem to agree that “the level of quality or the value provided to 
the government suffers” (Anderson, 2014, para. 2) when projects are awarded under 
LPTA. 

Industry often argues that not only does LPTA rob the government of long-term 
value by ignoring important non-cost factors, but that the level of customer service, 
overall effort, and quality in the short-run is also subpar. With all the profits “squeezed” 
out of the contractors, anything above the minimum requirement becomes unaffordable 
(Calisti, 2015). Anderson (2014) compared services received under LPTA to installing 
the cheapest tires for a car, “they may not last very long, and they could have blowouts 
that are not covered by warranties, they could cause vibrations at highway speeds due to 
imbalance but they will allow the car to roll down the road” (para. 2). Anderson (2014) 
also compared LPTA to achieving a “D” in school, which is the lowest passing grade, 
and encouraged govermnent contractors to “be diligent in reviewing the solicitation to see 
if the evaluation will be LPTA or Best Value” (para. 4); if it is LPTA, find out exactly 
what the “bare minimum” technical acceptable level is and strive for a D (Anderson, 
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2014). With such a stance, the defense industry believes that quality, creativity, and 
innovation have no place in a contract awarded under the LPTA method. 

3. LPTA Leaves No Room for Innovation 

Another consensus among industry with regard to increased use of LPTA is that it 
leaves no incentive for companies to innovate. With smaller profit margins, companies 
cannot afford to bring creative and innovative solutions to the table. In a Washington 
Technology article, Lisa Mascolo (2012) referred to LPTA as “the antithesis of 
innovative” (para. 3). Innovation plays a crucial role for both the DOD as well as the 
nation. During an interview, Zachary Fryer-Biggs (2012) captured a quote from the 
former U.S. defense secretary, Leon Panetta, on the importance of innovation: 

We’re depending a great deal on being at the technological edge of the 
future, Panetta said. I think we even have to leap forward. If we’re going 
to deal with the kind of challenges we’re going to face, we’ve got to be 
smart enough, innovative enough, creative enough to be able to leap 
forward, (para. 6) 

Innovation is also the heart of a nation’s economic development and evolution. In 
a world of global markets, innovation is also the key to obtaining competitive advantage. 
According to Gansler and Lucyshyn (2013), “it is clear that global markets; technology 
proliferation; and education in science, technology, engineering and mathematics will 
drive innovation in defense and other increasingly global industries” (pp. 21-22). 
However, Gansler and Lucyshyn (2013) worried that “increased reliance on LPTA may 
also accelerate worrisome national trends,” (p. 21) where “American dominance in 
scientific and technological innovation will continue to decline” (p. 22). Gansler and 
Lucyshyn (2013) further asserted that “reducing the demand for new innovation could 
potentially weaken America’s economy and national security posture” (p. 22). 

4. Impact of LPTA on the Defense Industry 

While defense spending has been cut across the board, the number of companies 
in the industry looking to win bids has not changed (Weckstein & Delgado, 2012). With 
less money to spread around, competition within the industry has been heating up over 

the past few years (Burnett, 2011). Additionally, the use of LPTA source selection 
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methods only seem to fuel the fire, further straining companies in the industry, forcing 
them to take drastic measures such as reducing employee salaries and layoffs (Perkins, 
2014). For instance, according to an article by Censer (2014), Wayne Luceroni, the 
president of Harris IT services, issued a statement during an interview that his company 
has cut costs, including consolidating seven Northern Virginia offices into one facility in 
order to “cope with what industry views as increasing price pressure” (para. 16). 
Furthermore, Young (2015) claimed that 

in this new LPTA business environment, savvy management teams have 
been reorganizing their business development teams and delivery 
capabilities around new standards. Operationally, contractors have 
stripped out all sorts of costs; reducing their wrap rates and overall cost of 
service delivery and product production. In terms of business 
development, many contractors have turned to outside consultants to assist 
with ad-hoc proposal development. Their efforts have yielded significantly 
lower Selling General and Administrative (SG&A) rates, which benefited 
the government financially on both fixed price and cost-plus programs. 

(P-1) 

Young (2015) also asserts that the outsourcing of proposal development by 
contractors is creating new problems for both government and contractor, where 
contractors are winning bids but unable to deliver performance because they did not write 
the proposals. As a response, the Department of Veterans Affairs (VA) banned proposal 
preparation by outside consultants on November 19, 2014, which was protested by a 
small business on grounds that the ban was “unduly restrictive” (p. 1). While the protest 
was denied by the GAO, “the implications for small and middle-market contractors are 
serious. If a company cannot afford to maintain full-time business development 
employees, it will be at a severe, if not insurmountable, disadvantage for competing on 
RFPs” (Young, 2015, p. 2). While the VA’s concerns over contract perfonnance risk are 
legitimate, the GAO’s decision would force smaller firms with limited resources into 
teaming agreements while larger primes benefit (Young, 2015). 

5. Leadership Awareness of LPTA Issues 

Proponents of LPTA believe that all the issues discussed thus far are just tactics 
used by industry to “demonize” the source selection method because it further narrows a 
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profit margin already impacted by budget cuts (Cox, 2015). Cox (2015) claimed that 
“contractors would prefer that DOD use a more expensive process called ‘best value,’ 
which allows them to charge more for features that warfighters usually don’t need,” 
(para. 4) and further insists that all the industry complaints published in trade reports are 
“unfounded” (para. 9). The biggest issue over the use of LPTA among industry is that it 
leads to low quality performance. According to Lohfeld (2012), the key to avoiding 
quality issues with the LPTA method is to ensure that the standards to meet the minimum 
technical acceptability is set at a level that meets the customer’s needs. Furthermore, cost 
and price analysis techniques such as price realism (price is too low) and price 
competitiveness (price is too high) can be utilized to evaluate proposals that seem too low 
or risky (Garrett, 2015). The controversy over the use of LPTA, however, is not ignored 
by Congress or the leaders in the DOD acquisition community. According to the House 
Armed Services Committee (HASC) Report 232-33 (as cited in Nichols, 2013), 

the House Armed Services Committee would have the U.S. Comptroller 
General conduct a review of [DOD’s] use of source selection processes, 
including LPTA-and, [i]n conducting the review, ... to obtain the views of 
defense contractors to gain insight into how the use of LPTA source 
selection procedures affects business decisions and to identify the 
unintended consequences, if any, resulting from the use of this approach. 

(P- 1) 

The HASC Report further “directs the Comptroller General to provide the 
findings of the review, along with recommendations to improve the Department’s 
contracting practices, to the congressional defense committees by June 30, 2014” (as 
cited in Nichols, 2013, p. 1). In addition to the Congressional inquiries, Jared Serbu 
(2014) reported that the USD(AT&L), Frank Kendall, tasked “Randall Culpepper, the Air 
Force’s top official for services contracting” (para. 3) to lead a review of LPTA usage 
across the DOD. 

G. SUMMARY 

This chapter explored general information on statutes and regulations that 
influence federal contracting and reviewed the phases of the contract management 
process. Additionally, the difference between LPTA and tradeoff source selection 


27 



methods were examined along with the concepts of best value and the best value 
continuum. The chapter also reviewed the new standardized DOD Source Selection 
Procedures, as well as some history on previous efforts to reform the federal acquisition 
process and best value principles. Finally, some controversial issues regarding the 
increased use of LPTA were discussed to set the stage for why this research is being 
conducted. The next chapter describes the methodology for this study, including 
background information regarding the Air Force Life Cycle Management Center, 
methods used to identify the data, and how the data was obtained. 
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III. METHODOLOGY 


This chapter will discuss the methodology used to locate, collect, and analyze data 
in order to answer the research questions. This includes a description of the data source, 
the types of data accessed, and the data collection process which describes the criteria 
used for selecting contract files and the data collection rubric. The last section describes 
the analysis protocol that was used to quantify the relationship (if any) between source 
selection method and contract outcomes such as CPARS ratings and PALT. 

A. SOURCE OF DATA 

For this research, we selected Wright-Patterson AFB as the source of data because 
of its perceived prominence in Air Force acquisition. Wright-Patterson AFB is home to 
both the Air Force Materiel Command (AFMC) and Air Force Life Cycle Management 
Center (AFLCMC) which are involved in the acquisition and sustainment of major 
weapons systems to support the Air Force mission. To most members of the Air Force 
acquisition community, Wright-Patterson represents the epicenter of procurement and 
acquisition activity. 

On June 4, 2010, a memorandum was issued by the Secretary of Defense, 
challenging services to “eliminate redundant functions and increase funding for mission 
functions through efficiency savings” (AFLCMC, 2015, P. 10). The Air Force believed 
the most prudent approach to meeting the SECDEF’s charge was a restructuring of the 
command. The decision was made that a reduction from 12 location-specific 
organizations, each requiring a full staff, down to five mission-based life-cycle centers 
would be the appropriate way ahead. The “Five Center Construct” as it was dubbed, was 
designed to “reduce overhead cost and eliminate redundant layers of staff’ (AFLCMC, 
2015, p. 11). It was also believed that such a structure would allow for more standardized 
business practices and provide customers with a “single face” for their programs 
(AFLCMC, 2015, p. 11). AFLCMC is one of the centers that fonned as a result of the 
restructuring. 
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The AFLCMC was designed to be a “mechanism to bring focused, disciplined 
and integrated life cycle management to the acquisition of Air Force weapon systems” 
(AFLCMC, 2015, p. 11). The overarching goal was to achieve savings and better 
performance by managing the entire life cycle of a system as opposed to its individual 
parts such as development, testing, production, and support. The belief was that a single 
center would provide unifonn and continuous process improvement at an enterprise level. 
The first AFLCMC commander, Lieutenant General Clyde D. Moore, developed the 
following mission statement for the new center: “Delivering affordable and sustainable 
war-winning capabilities to the U.S. and international partners on time, on cost, 
anywhere, anytime from cradle to grave” 

For this research, contracts from three of the 10 AFLCMC directorates were 
reviewed. The three directorates were the Agile Combat Support (ACS) Directorate, the 
Mobility Directorate, and the Contract Execution Directorate. ACS oversees Air Force 
programs to include metrology and calibration, automatic test systems, combat 
electronics, electronic warfare, and avionics and simulators (AFLCMC, 2015, p. 15). The 
ACS contracts reviewed for this study were all within the simulator section. The Mobility 
Directorate manages the “development, production, testing, deployment, and sustainment 
of a mobility and training aircraft” (AFLCMC, 2015, p. 17). Finally, the Contract 
Execution Directorate (PZI) office serves as an organization that provides contractual 
support to the various directorates spread across the AFLCMC and is primarily focused 
on providing infonnation technology solutions. 

B. TYPES OF DATA 

Numerous electronic federal acquisition systems exist, in part, due to federal 
statutes, regulations, policies, and in response to process improvement initiatives. These 
systems serve several functions including data collection, data dissemination, 
documenting contractor performance, and contract document execution. The systems 
accessed in support of this research include the Contract Writing System (ConWrite), the 
Federal Procurement Data System-Next Generation (FPDS-NG), the Contractor 
Performance Assessment Reporting System (CPARS), and the Earned Value Central 
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Repository. This section explains each system in further detail and highlights the reasons 
why they were accessed for this study. 

1. Contract Writing System 

ConWrite is one of several contract writing systems used by the United States Air 
Force (USAF). ConWrite is a contract document preparation software package that 
allows users to prepare contracts, solicitations, grants, modifications, and orders for Air 
Force Materiel Command (AFMC), Air Force Space Command (AFSPC) Product and 
Test Centers, and the Air Force Research Laboratory (B&ESD, 2015, p. 68). For this 
research, ConWrite was used to generate a report identifying contracts that met certain 
parameters. The report was then disseminated to several AFLCMC directorates to verify 
whether the contract files were available for review. 

2. Federal Procurement Data System-Next Generation 

FPDS-NG is “a computer-based Federal Procurement Data System for collecting, 
developing and disseminating procurement data to the Congress, Executive Branch, and 
private sector” (FPDS, 2015, p. 1). Information is entered into the system through a 
contract action report (CAR) once a contract action is completed. The system contains 
comprehensive post-award information pertaining to individual contracts awarded within 
the federal government. This information includes inputs such as contract type, the extent 
to which the contract was competed, and the number of offerors, among others. For the 
purposes of this research, FPDS-NG was used to generate a list of contracts that could be 
reviewed during the data collection process. 

3. Contractor Performance Assessment Reporting System 

CPARS is used to document contractor performance and “is one of the tools used 
to communicate contractor strengths and weaknesses to source selection officials and 
Contracting Officers” (CPARS, 2015, p. v). Contractor performance is assessed through 
five broad evaluation areas including Quality, Schedule, Cost Control, Management, and 
Utilization of Small Business, if applicable to the contract (CPARS, 2015). 
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CPARS reporting requirements differ between civilian federal agencies and the 
DOD. Although federal civilian agencies require CPARS reports for systems and non¬ 
systems contracts above the simplified acquisition threshold (SAT), the DOD typically 
requires CPARS reports for contracts at higher dollar thresholds (CPARS, 2015). 
Appendix A outlines the DOD CPARS requirements based on business sector and dollar 
threshold. 

The DOD CPARS requirements were useful to detennine search parameters for 
contracts in ConWrite and FPDS-NG. CPARS was accessed for this research because 
contractor performance data is an important post-award outcome. 

4. Earned Value Central Repository System 

Earned Value Management (EVM) is a program management practice to assess 
program status through cost, schedule, and technical performance (PARCA, 2015). 
Further, the Earned Value Management System (EVMS) “is an integrated management 
system that integrates the work scope, schedule, and cost parameters of a 
program in a manner that provides objective performance measurement data” 
(OUSD [ AT&L]/P ARC A, 2015). 

The DOD requirements for EVM are found in the Defense Federal Acquisition 
Regulation Supplement (DFARS) Subpart 234.2 and the DOD Instruction (DODI) 
5000.02 (OUSD[AT&L], 2015). The EVM reporting requirements are based on contract 
type and dollar threshold in then-year dollars (Appendix B). Referencing the EVM 
requirements was useful to determine search parameters for contracts in ConWrite and 
FPDS-NG. The Earned Value Central Repository was accessed for this research because 
EVM data is an important post-award perfonnance outcome. It is indicative of a 
contractor’s ability to manage the cost, schedule, and technical performance aspects of a 
program (PARCA, 2015). 

C. DATA COLLECTION 

The data collection process involved travelling to Wright-Patterson AFB to 
physically review contract files. Contracts were identified by working with AFLCMC to 
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run a query in both ConWrite and FPDS-NG based on particular parameters. This section 
explains the efforts made to identify and locate contract files and the spreadsheet rubric 
used to collect the necessary data. 

1. Criteria for Identifying Contract Files 

Reports were generated using ConWrite and FPDS-NG to identify contracts for 
review. Initially, AFLCMC Contracting Directorate (AFLCMC/PK) queried ConWrite 
based on the following criteria: (1) Contracts closed within the past calendar year (2015), 
(2) contract value >$1 million, and (3) ideal sample of 40-60 contracts. AFLCMC/PK 
was advised that if the initial report did not result in a list of 40-60 contracts based on the 
aforementioned criteria, an additional query could be accomplished to include all 
contracts that were within the last year of contract perfonnance (calendar year 2015). The 
resulting report included a total of 70 contracts that were closed within the past two years. 
This report was then forwarded to the following AFLCMC directorates to identify and 
locate any of the contract files: Business & Enterprise Systems (HI), Contract Execution 
(PZ), Air Force Security Assistance Cooperation (WF), Agile Combat Support (WN), 
645th Aeronautical Systems Group (WI), Mobility (WL), and Fighters and Bombers 
(WW). Unfortunately, many of the contracts listed in that report were unavailable for 
review because they were either in staging or they were noncompetitive contracts. 
ConWrite does not offer the ability to query based on competitive versus non-competitive 
contracts. To expand the potential amount of contracts to review, AFLCMC/PK provided 
an updated report that included all AFLCMC contracts that were awarded in calendar 
years 2010 and 2011. This report was much more extensive and included 910 contracts. 
The report was filtered and sorted for contracts valued greater than $1 million, which 
resulted in a list of 55 contracts. This report was also forwarded to the aforementioned 
directorates. Since reports generated from ConWrite could not identify whether contracts 
were competitive, a third report was generated using FPDS-NG. AFLCMC/AZ provided 
the report for all contracts awarded between FY2010-2013, which was then filtered by 
competitive contracts, resulting in a list of 188 contracts. All three reports were 
forwarded to the directorates to identify and locate contracts to review. 
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2 . 


Data Collection Rubric 


The data collection rubric is a Microsoft Excel spreadsheet created to capture up 
to 98 inputs per observation (contract reviewed). This is the same spreadsheet used in the 
studies conducted by Watson (2015) and Bastola, Findley, and Woodward (2015). The 
data inputs fell within five broad categories related to the pre-award and post-award 
contract management phases. The categories are: (1) Basic Information, (2) Acquisition 
Complexity, (3) Environmental Factors, (4) Outcome Variables, and (5) Other. This 
section describes each broad category in further detail. 

The Basic Information category includes descriptive infonnation for each 
observation such as the agency, the contract number, the name of the program or 
requirement, and the NAICS code. This category includes a total of seven data inputs. 

The Acquisition Complexity category includes variables such as the contract type, 
dollar value, source selection method (LPTA or tradeoff), number of evaluation factors, 
number of contract line item numbers (CLINs), number of offerors, and whether single or 
multiple award. This category included a total of 41 potential data inputs. 

The Environmental Factors category includes variables such as the number of 
reviews, number of personnel on the source selection team, the number of source 
selection team locations, and the use of collaboration tools among others. This category 
includes a total of 13 potential data inputs. 

The Outcome Variables category included the procurement administrative lead 
time (PALT) infonnation encompassing the requirement receipt date, RFP issue date, 
proposals received date, and contract award date. Additionally, other inputs include 
whether there was a protest (Yes or No), the level of the protest (if applicable), CPARS 
ratings, and EVM infonnation. This category includes a total of 30 potential data inputs. 

The Other category includes variables such as the number of evaluation notices, 
the number of rounds of evaluation notices, number of deficiency reports, and the basis 
for award. This category includes a total of seven potential data inputs. 
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D. DATA ANALYSIS 


This research aims to determine whether contract performance outcomes such as 
CPARS ratings, PALT, and EVM (if applicable) are impacted by the choice of using the 
LPTA or the tradeoff source selection method. This research utilized the same analysis 
technique used by Watson (2015) and Bastola et al. (2015). 

For this analysis, we used two main dependent, or outcome, variables (DVs) 1 : 
procurement administrative lead-time (PALT) and Contractor Performance Assessment 
Rating System (CPARS) data. PALT assesses “time to contract” by calculating the 
number of days between receipt of the requisition and contract award. It is a continuous 
variable. CPARS serves as a proxy measure of contract success or failure by using the 
ratings given to each contract once complete. CPARS ratings are given in Likert-style 
responses where l=Unsatisfactory, 2=Marginal, 3=Satisfactory, 4=Very Good, and 
5=Excellent. In this case, we used a combined CPARS score as the second dependent 
variable. Specifically, we searched for differences in CPARS ratings between LPTA and 
Trade-Off acquisition strategies by examining a within-contract average of CPARS Cost 
ratings, CPARS Quality ratings, CPARS Schedule ratings, and CPARS Business 
Relationship ratings. 

Our model has one independent variable, or IV. Independent variables are those 
that can be manipulated by the researcher (or user) to evoke a change in the outcome, or 
DV. In this case, the IV concerns the contract methodology used for the contract: lowest 
price technically acceptable (LPTA) or Trade-Off (TO). Contracting methodology is a 
choice made by the “user” (the integrated product team, which includes the Contracting 
Officer), hence it is considered an IV. Our IV is labeled LPTATO and it is a binary 
variable where 0=LPTA and l=Trade-Off. 

Finally, there are four covariate variables. Covariates are secondary variables that 
can also affect the relationship of primary interest: the relationship between the IV and 


1 EVM data was not available for the contracts that were reviewed. 
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the DV. Specifically, covariates are variables other than the independent variable 
that potentially affect the outcome variable, or DV. In this case, the covariates are (1) 
number of evaluation factors (NUMEVALF ACTORS), (2) number of reviews 
(NUMREVIEWS), (3) number of offers (NUMOFFERS), and (4) contract dollar value 
(VALUE).” All are continuous variables. Theoretically, each of the first three covariates 
could affect the PALT positively (e.g., a larger number of any of the covariates could 
increase the PALT, or time to contract). Each of the first three covariates may also have a 
positive effect on CPARS rating. One could argue that a higher number of each of the 
covariates serves to improve the chances of selecting the “right” or “best” contractor, 
ultimately improving contract perfonnance (as measured by CPARS rating). In this case, 
we hope to parcel out the effects of the covariates in order to more clearly see the effect 
the contracting methodology (LPTATO) has on the outcome variables (PALT and 
CPARS ratings). It is worth noting that not all of our covariates passed assumption 
testing, therefore not all were used. More details are provided in Chapter IV. 

E. SUMMARY 

This chapter discussed the methodology used to locate, collect, and analyze data 
in order to answer the research questions. This included a description of the data source, 
the types of data accessed, and the data collection process. The last section described the 
analysis technique that was used to quantify the relationship (if any) between source 
selection method and contract outcomes such as CPARS ratings and PALT. The next 
chapter will discuss the data analysis results. 


- One might assume that NUMREVIEWS and VALUE are highly correlated, given the fact that 
contract dollar value often determines the number and level of reviews through which a solicitation and 
contract must pass. However, for this data, we found they were not highly correlated (r=.03), thus we chose 
to keep both as covariates. 
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IV. DATA ANALYSIS 


This chapter describes the data analysis and results using multivariate and 
univariate analysis of covariance techniques to determine if there are differences in 
contract outcomes based on source selection methodology. The analysis is further 
supported by a discussion of the descriptive statistics for each variable, data issues, and 
assumption testing. 

A. DESCRIPTIVE STATISTICS 

Basic descriptive statistics for each variable are shown in Table 1. The table 
presents three figures for each variable: (1) the total for all the data, (2) the total for 
LPTA contracts, and (3) the total for tradeoff contracts. 


Table 1. Descriptive Statistics 


Variable 

Obs 

Mean 

SD 

Min 

Max 


PALT 

(days) 

23 

317.5652 

214.7235 

50 

878 

ALL DATA 

10 

203.1 

141.6054 

50 

482 

LPTA 

13 

405.6154 

223.9898 

72 

878 

TRADEOFF 

Average 

CPARS 

Rating 

18 

3.766667 

.639623 

3 

5 

ALL DATA 

5 

3.34 

.527257 

3 

4.2 

LPTA 

13 

3.930769 

.6179733 

3 

5 

TRADEOFF 

Number of 
Evaluation 
Factors 

24 

2.666667 

.5646597 

2 

4 

ALL DATA 

10 

2.1 

.3162278 

2 

3 

LPTA 

14 

3.071429 

.2672612 

3 

4 

TRADEOFF 

Number of 
Reviews 

22 

3.909091 

3.66332 

1 

18 

ALL DATA 

9 

2.555556 

2.068279 

1 

7 

LPTA 

13 

4.846154 

4.278749 

1 

18 

TRADEOFF 

Number of 
Offers 

24 

6.833333 

5.329709 

2 

23 

ALL DATA 

10 

8.1 

7.23341 

2 

23 

LPTA 

14 

5.928571 

3.452185 

2 

11 

TRADEOFF 

Contract 

Dollar 

Value 

24 

$268,000,000 

$595,000,000 

$1,185,396 

$2,300,000,000 

ALL DATA 

10 

$3,947,115 

$2,218,574 

$1,185,396 

$6,500,000 

LPTA 

14 

$456,000,000 

$732,000,000 

$1,271,044 

$2,300,000,000 

TRADEOFF 
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B. DATA ISSUES 


With only 24 cases, the sample size is small, and the distribution of cases by 
source selection method is unbalanced. Power calculations suggest the need for 14 cases 
for each source selection methodology (i.e., 14 LPTA cases and 14 tradeoff cases) in 
order to achieve adequate power (a = .05, P = .80). The data are unbalanced with respect 
to the number of cases for each source selection methodology. There are 10 LPTA cases 
and 14 tradeoff cases. This unbalanced design can cause ambiguity about the mean as the 
intercept and make assignment of sums of squares more difficult. There are, however, 
solutions to these issues. A weighted mean can be used in place of the grand mean and 
the STATA software automatically handles the assignment of the sums of squares. Thus, 
we proceeded with our analysis despite these issues. 

C. ANALYSIS 

Because our intent is to analyze differences in contract outcomes (PALT and 
CPARS ratings) based on source selection methodology (LPTA or tradeoff), a group 
comparison statistical methodology is necessary. In other words, the source selection 
methods are divided into two groups (LPTA and tradeoff), and we seek to find if there 
are differences in contract outcomes (PALT and CPARS ratings) by group. 

We used a technique called multivariate analysis of covariance, or MANCOVA, 
to assess group differences. MANCOVA essentially creates a new dependent variable 
using the information from the given dependent variables (PALT and CPARS rating). 
This new dependent variable is created in a way that maximizes differences between the 
grouping variable (LPTA or tradeoff source selections). MANCOVA addresses the 
following questions: Are mean differences among the groups on a combination of 
DVs (after adjusting for covariate effects) likely to have occurred by chance? Taken 
from another angle, is there a significant difference between the mean value for PALT 
and CPARS rating in the LPTA acquisitions versus the mean value for PALT and 


3 


The grand mean would be the intercept in a balanced design. 
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CPARS ratings in the tradeoff acquisitions, once the effects of the covariates 
(NUMEVALFACTORS, NUMREVIEWS, NUMOFFERS, VALUE) have been parceled 
out? MANCOVA examines the relationships between the dependent variables (PALT 
and CPARS ratings) and the independent variable (choice of LPTA or tradeoff 
methodology) while taking into account the effects the covariates might have on the 
dependent variables. 

If differences in outcomes are found using MANCOVA, researchers typically 
choose to dig deeper into the differences using a univariate technique called analysis of 
covariance (ANCOVA). ANCOVA essentially performs the same group differences 
analysis as a MANCOVA, however because it is univariate in nature, this method 
assesses one DV at a time (PALT or CPARS, rather than PALT and CPARS). This 
method helps researchers isolate where the difference is occurring (which IVs or 
covariates are affecting which DVs). We use both MANCOVA and ANCOVA in this 
research. 

D. ASSUMPTION TESTING 

Before conducting the MANCOVA, certain assumptions about the data were 
tested. First, we assessed multivariate nonnality by examining density graphs, 
determining multivariate skewness and kurtosis, and by examining the Doornik-Hansen 
test for multivariate normality (Doomik & Hansen, 2008). For the DVs, both PALT and 
CPARS were deemed to be non-normal. Both variables were normalized via a 
logarithmic transformation. All of the covariates (NUMEVALF ACTORS, 
NUMREVIEWS, NUMOFFERS, VALUE) also required a logarithmic transfonnation, 
after which all except NUMEVALFACTORS were deemed to be multivariate normal. 
We chose to drop the ill-behaved NUMEVALFACTORS covariate from all subsequent 
analyses, as “[cjovariates are often included as a convenience in reducing error, but it is 
hardly a convenience if it reduces power” (Tabachnick & Fidell, 2007, p. 251). 

Second, we searched for outliers using Mahalanobis’ Distance. We found no 
significantly influential outliers in our data. 
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Third, we assessed linearity by examining scatter plots of (1) the paired DVs, (2) 
all pairs of covariates, and (3) all pairs of DV-covariate combinations for each source 
selection method (LPTA and tradeoff, a total of 20 plots). The plots revealed that all 
relationships were linear in nature. 

Fourth, we assessed homogeneity of regression by performing an analysis of 
variance (ANOVA, which is practically the same as an ANCOVA but does not contain a 
covariate variable) that included the independent variable (LPTA or tradeoff), each of the 
covariates (NUMREVIEWS, NUMOFFERS, VALUE), and the interaction between the 
independent variable and the covariate. When the interaction terms are not significant, the 
relationship between the dependent variables (PALT and CPARS ratings) and the 
covariates is the same at both levels of the independent variable (LPTA or tradeoff), 
upholding the assumption of homogeneity of regression. With our data, significant 
interactions were found between (1) PALT and VALUE, (2) CPARS and VALUE, and 
(3) CPARS and NUMREVIEWS. Because VALUE failed the test of homogeneity of 
regression for both DVs, it was completely dropped from further analysis. Further, 
NUMREVIEWS could not be included in the MANCOVA, as it failed the homogeneity 
of regression test for the CPARS outcome variable. Thus, NUMOFFERS is the only 
covariate that is available for the MANCOVA. It is important to note that we use 
NUMREVIEWS in an ANCOVA (univariate) assessment of the relationship between 
source selection methodology (LPTA or tradeoff) and PALT, as NUMREVIEWS did not 
fail the homogeneity of regression test for the PALT outcome variable. 

Fifth, we checked for multicollinearity by assessing the pooled within cell 
tolerance for each DV (PALT and CPARS). Multicollinearity is not an issue for our data. 

Finally, we checked for homogeneity of covariance matrices between groups (i.e., 
sphericity) using the multivariate test of means provided in STATA 12.1. This test checks 
whether or not population variances and covariances of both dependent variables (PALT 
and CPARS) are equal for each of the independent variable groups (LPTA and tradeoff). 
The results showed that all grouping cells were homogenous. 
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With all assumptions tested, we performed the MANCOVA (with only 
NUMOFFERS as a covariate) and subsequent ANCOVA. The results are provided 
below. 

E. RESULTS 

1. MANCOVA 

The results show that source selection methodology (LPTA or Tradeoff) produced 
significantly different contracting outcomes at the p<.05 level. See Table 2. Specifically, 
we found that tradeoff source selections produce higher CPARS ratings (23% higher 
CPARS score). 

Breaking down the results further, we found that the NUMOFFERS submitted 
also serves to increase CPARS score (12% higher score for each percentage increase in 
NUMOFFERS, p<.10). Table 2 shows the results of this analysis. 


Table 2. MANCOVA Results—DVs: PALT and CPARS Ratings 


MANCOVA Results 

DVs: PALT and CPARS Rating 


Coeff 

SE 

t 

P>lt! 

95% Cl 

PALT 







LPTATO 

.4635959 

.3353186 

1.38 

.188 

-.255591 

1.182783 

# Offers 

.0856102 

.2357787 

0.36 

0.722 

-.4200848 

.5913052 

Constant 

5.336645 

.5807349 

9.19 

0.000 

4.091093 

6.582198 

CPARS 







LPTATO 

.2347992** 

.0885981 

2.65 

0.019 

.0447753 

.4248231 

# Offers 

.1194312* 

.0622976 

1.92 

0.076 

-.0141838 

-.2530462 

Constant 

.9350972 

.1534421 

6.09 

0.000 

.6059967 

1.264198 


* p<.10 ** p<.05 *** p<.01 

Number of Observations =17 

Root Mean Squared Error: PALT = .57 CPARS = .15 

R 2 : PALT = .1231 CPARS = .3585 


2. ANCOVA 

Given that two covariates (NUMREVIEWS and NUMOFFERS) passed the 
homogeneity of regression test for the outcome variable PALT, we decided to conduct a 
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post-hoc analysis to see if there was a significant difference in PALT based on source 
selection methodology (LPTA or tradeoff). Again, because we are only using one DV 
(PALT), we are using the univariate method (ANCOVA), not the multivariate method 
(MANCOVA). The results of the ANCOVA show that no differences in PALT exist 
based on source selection method. However, the covariate NUMREVIEWS significantly 
increases PALT (65% increase when using the tradeoff method, p<.01). This result is 
intuitive, as contract reviews add time to the procurement process and increasing the 
number of reviews will increase PALT. Typically, contract reviews are conducted 
consecutively and require approval before proceeding to the next step in the procurement 
process or to the next review. Table 3 shows the results of this analysis. 


Table 3. ANCOVA Results—DVs: PALT 


ANCOVA Results 

DVs: PALT 


Coeff 

SE 

t 

P>!tl 

95% Cl 

PALT 







#Reviews 

.6543624*** 

.1767935 

3.70 

0.002 

.2829331 

1.025792 

# Offers 

.1433843 

.1771919 

0.81 

0.429 

-.2288821 

.5156508 

LPTATO 

.1802472 

.2665075 

0.68 

0.507 

-.3796642 

.7401586 

Constant 

4.51685 

.3458353 

13.06 

0.000 

3.790277 

5.243422 


* p<.10 ** p<.05 *** p<.01 

Number of Observations = 22 

Root Mean Squared Error = .55236 

R 2 = 0.5551 

Adjusted R 2 = 0.4810 


Perhaps more telling are the relationships between the variables themselves. The 
correlations are given in Table 4. The lower triangle presents correlations of LPTA 
contracts, and the upper triangle presents correlations of tradeoff contracts. 
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Table 4. Correlation Data 


^Tradeoff 

LPTA^v 

PALT 

CPARS 

# Evaluation 
Factors 

# Reviews 

# Offers 

Value 

PALT 

1.00 

-0.6223 

0.6692 

0.7473 

-0.0580 

-0.0446 

CPARS 

0.6393 

1.00 

-0.4423 

-0.4888 

0.5523 

-0.3941 

# 

Evaluation 

Factors 

0.2671 

0.9118 

1.00 

0.9392 

-0.0940 

0.0388 

# Reviews 

0.7003 

0.9966 

0.8750 

1.00 

-0.0650 

-0.1648 

# Offers 

0.8953 

0.2298 

-0.1901 

0.3089 

1.00 

-0.6449 

Value 

0.9394 

0.8641 

0.5812 

0.9025 

0.6884 

1.00 


Variables with a correlation coefficient of +0.70 or greater indicates a strong 
positive linear relationship reaching perfect correlation at +1. Variables with a correlation 
coefficient of -0.70 or less indicates a strong negative linear relationship reaching perfect 
correlation at -1. A coefficient of 0 indicates that no relationship exists. 

a. LPTA Correlations 

CPARS and PALT have a moderately positive relationship at .6393. This suggests 
that taking additional time in the acquisition process (e.g., to craft the best solicitation 
possible, to ensure a thorough review of offers, to make a well-informed source selection 
decision, etc.) under an LPTA method leads to the selection of better performing 
contractors. Further, the almost perfect positive correlation between NUMREVIEWS and 
CPARS at .9966 indicates that more reviews lead to the selection of better perfonning 
contractors. 

NUMEVALFACTORS and CPARS have a strong positive relationship at 0.9118. 
This is an interesting correlation because it suggests that increasing the number of 
evaluation factors leads to the selection of better-performing contractors. Factors under 
an LPTA method are evaluated based on pass or fail basis. The more factors there are, the 
more aspects of the technical and non-price areas of the requirement that are evaluated; 
therefore, this relationship suggests better performing contractors are able to pass more 
hurdles in terms of evaluation factors than less qualified contractors. 
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NUMREVIEWS and PALT have a strong positive relationship of .7003. This is 
an expected relationship, because each review takes time and all reviews are typically 
consecutive. Even though requirements using LPTA are less complex than tradeoff, 
additional reviews will increase PALT. 

NUMOFFERS and PALT have a strong positive relationship of 0.8953. This 
result is also intuitive. The more proposals a source selection evaluation teams has to 
evaluate, the longer the PALT. Additionally, more offers could lead to increased 
exchanges with offerors through clarifications, communications, and additional rounds of 
discussions and resulting proposal changes, all of which serve to increase PALT. 

VALUE and PALT have a strong positive relationship of 0.9394. This is likely 
the result of the fact that higher value contracts typically require more reviews because 
many regulatory procedures are based on, or triggered by, dollar thresholds. The strong 
positive relationship of .9025 between Value and the number of reviews also supports 
this relationship. 

b. Tradeoff Correlations 

Variables for tradeoff contracts did not exhibit similar correlation patterns as 
LPTA contracts. Overall, tradeoff variables exhibited more negative relationships. 

CPARS and PALT have moderately negative relationships of -0.6223. It is not 
clear why an increase in PALT would lead to a lower CPARS rating and vice-versa. 
Additionally, NUMREVIEWS and CPARS also has a negative relationship (r = -.4888). 
These results are opposite of the results observed under the LPTA method for the same 
relationships. It is unclear why this difference exists. 

NUMEVALFACTORS and PALT have moderately positive relationships of 
0.6692. This is likely an indication that increasing the number of evaluation factors 
increases the amount of time it takes the source selection team to complete proposal 
evaluations, thus increasing PALT. NUMREVIEWS also had a strong positive 
relationship with PALT of 0.7473, which is similar to the correlation for LPTA contracts. 


44 



This is expected because contract reviews typically increase the PALT regardless of the 
chosen source selection method. 

Surprisingly, NUMEVALFACTORS and CPARS and NUMREVIEWS and 
CPARS have low negative relationships of -0.4423 and -0.4888, respectively. This is 
contrary to the strong positive relationship between these variables for LPTA contracts. 
These results are counterintuitive, as one would theoretically expect a higher number of 
evaluation factors and reviews to yield a better-perfonning contractor. More data is 
needed to detennine the stability and significance of these negative relationships. 

NUMOFFERS and CPARS have moderately positive relationships of 0.5523. 
This is intuitive based on the expectation that more offers provides the SSA with more 
options to choose from, which leads to a better chance of selecting a high-performing 
contractor. 

Our analysis presented some unexpected results for the tradeoff correlations. 
More data and further analysis are necessary to check the stability of these results. 

F. SUMMARY 

This chapter discussed the data analysis and results using MANCOVA and 
ANCOVA techniques. It also included some post-hoc analyses which looked closer at the 
relationships between covariates and contract outcomes. An examination of the 
relationships between some of the covariates and contract outcomes identified areas for 
further research. The next chapter will provide our conclusions and recommendations 
based on our findings. 
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V. CONCLUSIONS 


This chapter summarizes the purpose, analysis, and findings of this research, and 
answers the research questions posed in Chapter I. The chapter concludes with 
recommendations for further research. 

A. SUMMARY 

As stewards of taxpayer dollars, government acquisition teams have an obligation 
to provide the best value for products and services required by their customers. While 
conducting a successful source selection is a critical part of the acquisition process, the 
ability to achieve best value is affected by the activities performed during all the phases 
of the federal contract management process. Within the DOD, LPTA and tradeoff are the 
two most widely used source selection methods on the best value continuum. The trend 
for utilizing LPTA in the DOD increased over the past few years along with sharp 
criticism of the method by the defense industry. Contractors maintain that the DOD is 
using the LPTA method inappropriately and that it robs the government of high quality 
products and services and innovative solutions. The GAO disputes these claims, offering 
report findings that assert the DOD is using the method appropriately. Currently, there is 
no empirical evidence that li nk s the utilization of the LPTA source selection method to 
poor contactor perfonnance outcomes or the use of the tradeoff method to good 
performance outcomes. The objective of this research was to determine whether the 
source selection method employed by the Air Force, either LPTA or tradeoff, affects the 
time-to-contract metric (PALT) and/or the contractor’s performance (as measured by 
CPARS). 

B. CONCLUSION 

Based on our results, we can now answer the research questions that were 
presented in Chapter I: 

1. What is the relationship between a particular source selection method and 
the resulting performance outcome of the contractor? 
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For the first question, our analysis supports that a relationship exists between a 
particular source selection method (LPTA or tradeoff) and the resulting contractor 
performance outcome. We found that source selections conducted under the tradeoff 
method produced higher perfonnance outcomes as measured by CPARS. Additionally, 
we found that there is a positive relationship between the number of offers submitted and 
the perfonnance outcome of the contractor as measured by CPARS. In other words, the 
more competitive the acquisition, the better the performance outcome of the selected 
contractor. 

2. What is the relationship between a particular source selection method and 
the PALT? 

i. If there is a positive relationship, can the amount of additional 
resources utilized for that particular source selection method be 
quantified in terms of man-hours or in dollars? 

For the second question and its first sub-question, our analysis does not support a 
relationship between a particular source selection method and the PALT. Nonetheless, 
there are covariate factors that can affect the PALT. For example, the number of reviews 
positively impacts the PALT under both LPTA and tradeoff source selection methods, 
which means that the more reviews there are, the longer time it takes to get to award. As 
a second example, the number of offers affects the PALT positively when using LPTA 
but slightly negatively when using tradeoff. More specifically, the greater the number of 
offers to evaluate under an LPTA source selection, the longer it takes to get to award, yet 
greater numbers of offers under a tradeoff source selection slightly reduces time to award. 
Clearly, this ambiguous finding requires further research to disentangle. 

ii. Does the resulting perfonnance outcome merit the use of 
that particular source selection method? 

Since our research seems to support the fact that a contractor selected through the 
tradeoff method has higher performance outcomes than under LPTA, the answer to this 
sub-question is yes, the tradeoff method produces better perfonnance, based on CPARS 
scores. Moreover, since PALT is not affected by the source selection method, there is no 
reason to shy away from the use of the tradeoff method to achieve better contractor 
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performance. Clearly, these results are specific to our data, and the collection of 
additional data is necessary to confirm these findings. 


C. RECOMMENDATIONS FOR FURTHER RESEARCH 

This research project produced some interesting results with data collected from 
the AFLCMC, which is only one center within an Air Force MAJCOM (AFMC) known 
for its prominence in acquisitions. We recommend that additional data be collected and 
analyzed to confirm the findings of our research. Furthermore, due to time and manpower 
constraints, and the fact that the AFLCMC customarily engages in sole source contracts, 
it was difficult for us to collect a well-balanced and statistically powerful amount of data. 
We recommend repeating this research in its entirety with a larger sample size and data 
representing all the organizations in AFMC. 

Since the AFMC consists of mostly systems level procurements, it would be 
beneficial to repeat this research across the Air Force and other agencies to assess the 
outcomes for different types of requirements across the DOD. The results may vary 
according to the type of requirement (i.e., construction, services, and commodities). 
Understanding these relationships across the various requirement types can help 
acquisition teams plan their acquisition strategy in a way that cuts waste and streamlines 
the acquisition process while still achieving best value. 

Our final recommendation is to devise a research project focused on quantifying 
the amount or level of resources consumed during the course of a procurement process 
from beginning to end. We expected to find a significant relationship between source 
selection method and PALT due to the fact that average PALT for tradeoff is double the 
average PALT for LPT A, however our results did not support such a relationship. 
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APPENDIX A. BUSINESS SECTOR, DOLLAR THRESHOLD 
AND REVIEWING OFFICIAL (FROM CPARS, 2015) 



Dollar Threshold 1 

2 

Reviewing Official 

Civilian Agencies (excludes 
DOD): 



Systems and Non-Systems 

>Simplified 

Acquisition 

Threshold 

One level above the contracting 
officer, as determined by department 
or agency policy 

Architect-Engineer 

>$30,000; 

All Terminations 
for Default 

One level above the contracting 
officer, as determined by department 
or agency policy 

Construction 

>$650,000; 

All Terminations 
for Default 

One level above the contracting 
officer, as determined by department 
or agency policy 

DOD Services and Agencies: 
Systems (includes new 
development and major 

>$5,000,000 

One level above the PM -5 

Non-Systems 

Operations Support 

>$5,000,000 4 

One level above the AO 

Services 

>$1,000,000 

One level above the AO 

Information Technology 

>$1,000,000 

One level above the AO 

Ship Repair and Overhaul 

>$500,000 

One level above the AO 

Architect-Engineer 

>$30,000; 

All Terminations 
for Default 

One level above the AO 

Construction 

>$650,000; 

All Terminations 
for Default 

One level above the AO 


The contract/order thresholds for CPARS collection (see FAR 42.1502) apply to the “aggregate” value of 
contracts/orders; that is, if a contract’s/order’s original award value were less than the applicable threshold but 
subsequently the contract/order was modified and the new value is greater than the threshold, then evaluations are 
required to be made, starting with the first anniversary that the contract’s/order’s face value exceeded the threshold. 

If the total contract/order value including unexercised options and orders (for IDIQ contracts, total estimated value 
of unexercised options and orders) is expected to exceed the collection threshold, initiate the collection process at 
the start of the contract/order. Buying activities may choose to collect perfonnance evaluations for awards below 
these thresholds. 

2 

Only required when the contractor indicates non-concurrence with the CPARS or if otherwise requested 
by the contractor during the 60-calendar day comment period. 

3 

(Or equivalent individual) responsible for program, project or task/job order execution. 

4 

For contracts/orders under the reporting thresholds, buying activities should continue to accumulate 
contractor perfonnance data from existing management information systems, which already capture data 
on timeliness of delivery and quality of product or service (an example of a performance infonnation 
collection systems is the Past Perfonnance Information Retrieval System—Statistical Reporting Module) 
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APPENDIX B. EVM REQUIREMENTS (FROM OUSD[AT&L], 2015) 


REQUIREMENTS 

WHEN REQUIRED 1 

SOURCE 

For Cost/Incentive Contracts 2 > $50 Million 3 

Compliance with EVM 
system guidelines in ANSI/EIA- 

At contract award and throughout 
contract performance 

Part 7 of Office of 
Management and 

Budget (OMB) Circular 

A-11 (Reference ©) 

This instruction 

EVM system formally validated 
and accepted by cognizant contracting 

At contract award and throughout 
contract performance 

Integrated Program 

Management Report (IPMR) (Dl- 

Monthly 

Integrated Baseline Reviews 

Within 180 calendar days after contract 
award, exercise of options, and major 

For Cost/Incentive Contracts 2 > $20 Million 3 but < $50 Million 3 

Compliance with EVM 
system guidelines in ANSI/EIA- 
748 4 

At contract award and throughout 
contract performance 

Part 7 of OMB Circular A- 
11 This instruction 

- IPMR (DI-MGMT-81861 5 ) 
(tailoring of formats recommended) 

Monthly 

Integrated Baseline Reviews 

For Cost/Incentive Contracts 2 < 

$20 Million 3 

Within 180 calendar days after contract 
award, exercise of options, and major 

At the discretion of the Program Manager 
based on cost-benefit analysis 

Part 7 of OMB Circular A- 
11 This instruction 

- IPMR, Format 6 (DI-MGMT- 

For Firm Fixed-Price 

Contracts 2 regardless of 

At the discretion of the Program Manager 
hased on Government renuirements 

Limited Use-will be approved by the MDA 
based on a business case analysis 

Part 7 of OMB Circular A- 
11 This instruction 

- IPMR, Format 6 (DI-MGMT- 

At the discretion of the Program Manager 
based on Government reauirements 

Notes: 

1. EVM is required, as outlined in the table, unless the EVM requirement has been waived by the CAE per paragraph 6c in Enclosure 

2. 

2. The term, “Contracts,” includes contracts, subcontracts, intra-government work agreements, and other agreements. For Indefinite 
Delivery/Indefinite Quantity contracts, EVM will be applied to the individual task orders or group of related task orders in accordance with 
the requirements in this table. “Incentive” contracts include fixed-price incentive. EVM is required for Fixed-Price Incentive Fee 
development and integration contracts with measurable and discrete work scope. In cases where the work scope is not measurable 
and discrete, program offices should follow the process to obtain a DFARS deviation. 

3. Application thresholds are in then-year dollars. 

4. ANSI/EIA-748 = American National Standards Institute (ANSI)/Electronic Industries Alliance (EIA) Standard 748, Earned 

Value Management Systems (Reference (au)). 

5. DI-MGMT-81861 = DID: Integrated Program Management Report (Reference (av)) 

a. If EVM is not required or a deviation is obtained, the IPMR should be used and tailored to obtain cost and/or schedule reporting 
when desired by the Government. For example, for full rate production contracts where EVM is not applicable, a tailored IPMR 
including a cost report showing actuals and a top-level schedule providing delivery dates of end products would be sufficient for 
Government management and oversight. 

b. Flow-down of the IPMR DID to the subcontractors is at the discretion of the program office. 
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